Papers with multimodal representation learning
ViLMedic: a framework for research at the intersection of vision and language in medical AI (2022.acl-demo)
Copied to clipboard
Jean-benoit Delbrouck, Khaled Saab, Maya Varma, Sabri Eyuboglu, Pierre Chambon, Jared Dunnmon, Juan Zambrano, Akshay Chaudhari, Curtis Langlotz
| Challenge: | Multimodal medical AI is a growing field of interest, especially for tasks that involve multimodal data. |
| Approach: | They propose a vision-and-language medical library to improve multimodal medical predictions and enable new applications. |
| Outcome: | The vision-and-language medical library aims to improve reproducibility and speed up progress across medical AI . it contains a dozen implementations replicating state-of-the-art results on medical datasets . the library is extensible by researchers but also simple for practitioners . |
ColMate: Contrastive Late Interaction and Masked Text for Multimodal Document Retrieval (2025.emnlp-industry)
Copied to clipboard
Ahmed Masry, Megh Thakkar, Patrice Bechard, Sathwik Tejaswi Madhusudhan, Rabiul Awal, Shambhavi Mishra, Akshay Kalkunte Suresh, Srivatsava Daruru, Enamul Hoque, Spandana Gella, Torsten Scholak, Sai Rajeswar
| Challenge: | Existing methods for multimodal document retrieval often replicate techniques developed for text-only retrieval. |
| Approach: | They propose a document retrieval model that bridges the gap between multimodal representation learning and document retrievals by providing external knowledge as context. |
| Outcome: | The proposed model achieves 3.61% improvement over existing retrieval models on the ViDoRe V2 benchmark, showing stronger generalization to out-of-domain benchmarks. |
Compressing then Matching: An Efficient Pre-training Paradigm for Multimodal Embedding (2026.acl-long)
Copied to clipboard
Da Li, Yuxiao Luo, Keping Bi, Jiafeng Guo, Wei Yuan, Biao Yang, Yan Wang, Fan Yang, Tingting Gao, Guorui Zhou
| Challenge: | Recent approaches demonstrate that MLLMs can be adapted into competitive embedding models via large-scale contrastive learning. |
| Approach: | They propose a compressed pre-training phase which serves as a warm-up stage for contrastive learning. |
| Outcome: | The proposed model achieves state-of-the-art among MLLMs of comparable size on the MMEB, realizing optimization in both efficiency and effectiveness. |
Knowledge-enhanced Multimodal ECG Representation Learning with Arbitrary-Lead Inputs (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Current methods for multimodal representation learning for electrocardiograms often result in suboptimal alignment of ECG signals with their corresponding text reports. |
| Approach: | They propose a framework to learn ECG representations by aligning ECG signals with paired free-text reports. |
| Outcome: | The proposed framework outperforms existing methods in zero-shot classification and linear probing tasks using 12 leads. |
Unimodal and Crossmodal Refinement Network for Multimodal Sequence Fusion (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to modulate one modal feature to another are lacking in multimodal representation learning. |
| Approach: | They propose to use unimodal and crossmodal refinement networks to enhance uni and cross-modal representations by iterative updating of distributions with transformer-based attention layers to refine modality-specific learning. |
| Outcome: | The proposed network outperforms state-of-the-art techniques on MOSI and MOSEI datasets. |
Evaluating Perspectival Biases in Cross-Modal Retrieval (2026.findings-acl)
Copied to clipboard
Teerapol Saengsukhiran, Peerawat Chomphooyod, Narabodee Rodjananant, Chompakorn Chaksangchaichot, Patawee Prakrankamanant, Witthawin Sripheanpol, Pak Lovichit, Sarana Nutanong, Ekapol Chuangsuwanich
| Challenge: | a recent study shows that multimodal retrieval systems are expected to operate in a semantic space, agnostic to the language or cultural origin of the query. |
| Approach: | They introduce a benchmark to quantify linguistic and cultural biases in multimodal retrieval systems . they propose a framework to decouple language from culture and decouples it from semantics . |
| Outcome: | The proposed benchmark systematically measures the effects of linguistic and cultural biases on retrieval performance. |